1 "AUTOMATIC COLOR ADJUSTMENT FOR DIGITAL IMAGES" 

2 CROSS REFERENCE TO RELATED APPLICATION 

3 This application is a continuation in part of pending US Patent 

4 application Serial No. 09/548.755 entitled - VIRTUAL TRUE COLOR LIGHT 

5 AMPLIFICATION, filed - April 13, 2000 and which claims priority of US 

6 Provisional Patent application 60/129,041. filed April 13, 1999. The entire 

7 disclosures of these applications are incorporated by reference herein in their 

8 entirety. 

9 FIELD OF THE INVENTION 

10 The present invention related to methods for enhancing digital color 



1 1 images. More particularly, the method automatically selects the extent to which the 

12 strength of light is amplified in con-ecting color digital images without exceeding the 

13 inherent dynamic range for the image. Further, an automated method corrects 

14 faded or washed out images. 

15 



16 BACKGROUND OF THE INVENTION 

17 LIGHT AMPLIFICATION OF IMAGES 

18 If one were to photograph the stain glass windows of a Cathedral, the 

19 resulting image would be typically too dark for modern tastes. The beauty of the 

20 actual stained glass is lost. Further, a photographer can increase depth of field of 

21 an image by reducing the device's aperture, but light collection is sacrificed and can 

22 also result in a dark image. Further, as a still image device is capable of only 
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1 HSV color space, Kuo then isolates and removes the color information (Hue) from 

2 the remaining image components (saturation and intensity). Kuo suggests that the 

3 components of saturation and intensity can be enhanced without introducing 

4 distortion into the color or Hue component. Kuo's color image is represented by a 

5 plurality of pixels in HSV color space. Once transformed, Kuo inverse transforms 

6 HSV back to RGB color space, all the while claiming this to be efficient. In the 

7 preferred embodiment, Kuo adjusts intensity (V) and saturation (S). In summary, 

8 Kuo first transforms RGB to HSV color space, applies two sequential transformation 

9 functions to V and S respectively, and finally inverse transforms the altered HSV 

1 0 back to RGB for display. 

11 Respectfully, Applicant asserts that manipulation of the saturation 

12 does affect the color and thus the Kuo technique does not result in true color 

13 enhancement. Color photos have three degrees of freedom, being R,G and B, 

14 where as black and white photos only have one. Any transformation of the three 

15 values results in three more values, each of which contains a color component and 

16 not merely a single color value (i.e. hue) and two other independent structural 

17 components (i.e. saturation and intensity). Hue may be a more extreme sense of 

18 color than saturation, but saturation is still a color component. Adjusting a dot's 

19 saturation results in a change in the proportions of RGB in the dot and hence its 

20 color. If the saturation is changed, say as part of brightening the image, the result is 

21 not the same as if the recording device or camera had obtained the image directly 

22 from the real world subject under brighter conditions, or more exposure. 
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1 Further, note that Kuo emphasizes and attempts to minimize the 

2 computational overhead or expense. Unfortunately, Kuo introduces two RGB-HSV 

3 and HSV-RGB transformations in addition to whatever adjustments (preferably two) 

4 Kuo makes to the HSV pixel. A transformation from RGB to HSV color space, and 

5 back again, involves the use of computation-intensive mathematical functions. 

6 Further, the foregoing methodologies rely on significant user input and 

7 skill. 

8 

9 AUTOMATIC CORRECTION 

10 It is known to automatically corrects tone values of digitally stored 

1 1 images as disclosed in US patent 5,544,258 to Levien. A histogram of the image is 

12 processed by adding a constant, applying a non-linear function to each value, and 

13 digital filtering. Application of a square root function assures that small histogram 

14 values have more contribution to the final curve, and that large histogram values 

15 have less contribution to the final tone curve, relatively. The processed histogram is 

16 integrated and then normalized for a tone correction function. This tone correction 

17 function can be applied to the average histogram of the respective individual color 

18 planes. It is applicant's view that this correction does not maintain the true colors 

19 through the correction nor deal with limitations of dynamic range. 

20 Accordingly, there is demonstrated a need for an automatic and 

21 computationally efficient process which is capable of maintaining true color for each 
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1 dot during enhancement of an image. Automation avoids the barrier of a steep 

2 learning curve to effectively adjust images for more a pleasing effect. The present 

3 invention addresses these problems by providing a technique which is both 

4 automated and which, no matter how much image-brightening is needed or what 

5 the nature of that brightening is, the color of all dots in the image are preserved. 

6 

7 SUMMARY OF THE INVENTION 

8 The present invention applies the virtual true color correction 

9 techniques of the co-pending application in an automated methodology for further 

10 simplifying color light amplification of digital photographs without color distortion. In 

1 1 one embodiment, the present invention automatically applies an average strength 

12 value for a digital image and determines a corresponding correction function for 

13 maximal amplification without color distortion. In another embodiment, the method 

14 is further improved through automatic compensation for distorted or faded digital 

15 photographs. 

16 In one broad aspect of the invention, automatic adjustment of the color 

17 of a digital image comprises: establishing an original histogram of dot maximums for 

18 the plurality of color dots within the original image and an original average strength 

19 thereof; amplifying each dot maximum with a scaling factor selected from a 

20 continuous scaling function to obtain a scaled dot maximum which is less than or 

21 equal to the maximum of the dynamic range; establishing a conrected histogram of 

22 scaled dot maximums and a conrected average strength thereof; interpolating a 
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1 target scaling factor from a target average strength, the conrected average strength 

2 and the original average strength for each dot maximum; and creating the adjusted 

3 image without color distortion by applying the target scaling factors to each color 

4 dot's R, G and B strength values so that a histogram of the adjusted image has the 

5 target average strength and that the ratios of the strength values between R, G and 

6 B for the color dot remain the same after scaling as they were before scaling. 

7 In some cases it is preferable to automatically correct the image for 

8 removing excessive gray before automatic adjustment of the color further 

9 comprising: establishing a supplied histogram of R, G, or B strength values for each 

10 color dot within the supplied image; determining a offset strength value between the 

1 1 minimum of the dynamic range of the system and a minimum threshold strength 

12 value of the supplied histogram; and subtracting the offset strength value from each 

13 R, G, and B value for forming the original image for automatic correction thereof. 

14 
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1 BRIEF DESCRIPTION OF THE DRAWINGS 

2 VIRTUAL TRUE COLOR LIGHT AMPLIFICATION 

3 In the Figures, several digital images are presented. As this 

4 application is directed to the enhancement of color digital images without suffering a 

5 distortion in color, the results cannot be properly reproduced herein in the gray 

6 scale printing medium. 

7 Figure 1 is a graph representing a linear function as the basis for 



8 correcting each R,G or B dot maximum as input to an adjusted output dot 

9 maximum, both of which are constrained to the system's dynamic range. This 

10 particular function would be a unity function, and would not perform any correction 

1 1 unless the input is normalized to the dynamic range by pre-scaling the maximum of 

1 2 the dot maximums to 1 .0; 



13 Figure 2 is a brief coding example in Visual Basic for reading a digital 

14 screen image, extracting color dots, finding a dot maximum, applying a correction 

15 factor for the dot maximum, applying the correction to the entire RGB values for a 

16 dot and writing the corrected color dot back to the screen; 

17 Figure 3 is a graph illustrating a scaling function designed to modify an 

18 image which was intentionally underexposed, such as by using a small aperture so 

19 as to obtain improved depth of field; 

20 Figure 4 is a graph illustrating a scaling function which enhances the 

21 contrast within a specific area of the dot maximums, falling between 0.3 - 0.5 of the 
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1 image's range, by scaling 20% of tlie range to nearly 100% or substantially the 

2 entire dynamic range, and wherein the darker and lighter areas contrasts are 

3 diminished; 

4 Figure 5 is a graph according to Fig. 4 which enhances the dot 

5 maximums in the dark area falling between 0.1 - 0.2 of the image's range; 

6 Figure 6 is a graph according to Fig. 4 which enhances the dot 

7 maximums in the bright area falling between 0.9 - 1 .0 of the image's range; 

8 Figure 7a is a brief coding example in Visual Basic for using a GUI 

9 interface to select an x1 ,y1 and x2,y2 window area, reading the digital screen image 

10 in the window, extracting color dots, finding a dot maximum, applying a correction 

1 1 factor for the dot maximum, and building a histogram of dot maximum occurrences; 

12 Figure 7b is a brief coding example in Visual Basic for generating the 

1 3 histogram according to the third embodiment. 

14 Figure 8 is a graph illustrating variable scaling functions superimposed 



15 over a unity diagonal, the variable functions produce an aesthetically pleasing 

16 enhancement through the brightening of the image. The scaling functions are 

17 smooth curves, one a factor of the other, such as a third order curve, which de- 

18 emphasizes the darker areas and brightens the lighter areas; 

19 Figures 9a - 9f are photographs of an Abbey which are respectively, 

20 the original, brightened under the prior art, brightened and contrast adjusted under 

21 the prior art, brightened and saturation adjusted under the prior art, enhanced 
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1 according to the first embodiment of the present invention to the full dynamic range 

2 and enhanced according to the second embodiment of the present invention; 

3 Figures 10a - 10f are photographs of Stone Henge which are 

4 respectively, the original, brightened under the prior art, brightened and contrast 

5 adjusted under the prior art, brightened and saturation adjusted under the prior art, 

6 enhanced according to the first embodiment of the present invention to the full 

7 dynamic range and enhanced according to the second embodiment of the present 

8 invention; 



9 Figures 11a and lib are respectively an original photo of a satellite 

10 and an enhanced photo according to the third embodiment of the present invention; 

1 1 Figures 12a and 12b are respectively an original photo of a blimp and 

12 an enhanced photo according to the third embodiment of the present invention; 

13 Figures 13a and 13b are respectively an original photo of a car license 

14 plate and an enhanced photo according to the third embodiment of the present 

15 invention; 

16 Figures 14a, 14b and 14c are respectively an original photo of skiers 

17 and ski tracks in the snow and two enhanced photos according to the third 

18 embodiment of the present invention, each using a different portion of the photo to 

19 build the enhancement; and 

20 Figure 15a is a flow chart of one embodiment of the invention 

21 illustrating determination of scaling factors; 
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1 Figure 15b is a flow chart of another embodiment of the invention 

2 illustrating determination of scaling factors and storing them in a lookup table; 

3 Figure 15b is a flow chart of another embodiment of the invention 

4 illustrating determination of scaling factors and application of a function to adjust the 

5 color dots; 

6 Figure 16 is a flow chart of another embodiment of the invention 

7 illustrating a normalization process for the dots to at least a portion of the dynamic 

8 range such as that set forth in the third embodiment; and 

9 Figure 17 is a flow chart of another embodiment of the invention 

10 illustrating a selection of a portion of the image for adjustment and two 

1 1 implementations of the invention to adjust the image; 

12 Figures 18a and 18b are respectively a first histogram of a supplied 

13 image which is faded or has an excessive gray present, and a second histogram of 

14 an image which has been offset to remove gray according to an automatic 

1 5 correction embodiment of the present invention; 

16 Figure 19 is a graph according to Fig. 8 further illustrating a variety of 

17 scaling functions superimposed over an identity function or unity diagonal, one non- 
18 linear curve representing a nominal correction which can be adjusted to an 

19 interpolated target function which better produces an aesthetically pleasing 

20 enhancement through the brightening of the image; and 
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1 Figure 20 illustrates three comparative histograms including a second 

2 histogram Ho of an offset image forming an original image for brightening according 

3 to Fig. 18b, a third histogram He of the connected image at 100% application of the 

4 scaling function C according to Fig. 19, and a fourth histogram Ht of a final 

5 conrected image at the interpolated and fractional correction of the scaling function 

6 T to achieve a target average strength. 

7 
8 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 



2 VIRTUAL TRUE COLOR LIGHT AMPLIFICATION 

3 The present invention disclosed herein implements a new method for 

4 automating techniques for amplifying the brightness of color digital images, the 

5 basics for brightening images being are set forth in co-pending application Serial 

6 No. 09/548,755 to the applicant, the entirety of which is incorporated herein by 

7 reference. A substantial portion of the co-pending application is reproduced herein. 

8 First, an image is captured using some form of digital image recorder. 

9 Digital image recorders fall into two categories: physical and virtual. Physical digital 

10 image recorders are devices that record a digital image by the measurement of light 

11 energy; such as a digital cameras. Like a traditional film camera, digital cameras 

12 have a 'lens complex' that provides light gathering and the image is recorded by an 

13 array of digital sensors so that the value of each dot represent actual 

14 measurements of the light. A digital image can also be obtained as a digital scan of 

15 a traditional photograph. In a photograph, light gathering was provided by a 

16 traditional camera and the image was recorded on film. Accordingly, a physical 

17 digital image recorder can be a combination of camera that produced the film/print 

18 image and a scanner that digitized it. Other examples include digital movies, 

19 digitized movies, digital x-rays, and the like. 

20 Virtual digital image recorders are computer renderings that imitate 

21 reality. The programs create a 'virtual image' (as in virtual reality) by a logical 

22 imitation of the photographic process completely internal to the computer itself. 

12 



1 These digital images are what a photograph would have looked like had a 

2 'computer model' actually existed. An example is those movies with stunning 

3 dinosaur simulations. 

4 A lens complex is the apparatus that gathers light in various forms of 

5 photography. There is at least one lens and usually a system of such lenses. The 

6 lens complex also includes an aperture stop and a shutter which are both controls 

7 on the light gathering. 

8 The light gathering power of a lens is often measured in terms of the 

9 surface area of the objective lens itself. A lens with twice the area of another can 

10 gather twice the light. In practice, a lens complex has two controls on the amount of 

1 1 light actually gathered. The first control is the adjustable aperture which varies the 

12 amount of light that is collected per unit time. Twice the area means twice the light 

13 per unit time. The second control is called the shutter and it varies the amount of 

14 time that light enters the body of the camera. Holding the shutter open twice as 

1 5 long means that twice the light energy enters the camera. 

16 A true color digital Image comprises a grid of dots wherein each dot 

17 has three independent measured values representing the strengths of the red, 

18 green and blue (RGB) components of the light. This is known as RGB color space. 

19 There are various computer file formats used for these images, using various 

20 'compression schemes' to save computer disk storage space. Regardless of 

21 compression scheme, all such digital file formats store a grid of dots with RGB 

22 values. 
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1 Currently, the common standard maximum value stored by such files 

2 or a system, for each of the R,G, or B value, is 255. Accordingly, each of the RGB 

3 components can range In strength from 0 to 255. Some file formats now store 

4 values in the range of 0 to 1023, and higher formats are likely. 

5 

6 Dynamic ranee 

7 Every device, including our digital image recorder, has a 'dynamic 

8 range' which is a measure of its ability to record relative energy fluctuations. As 

9 stated above, this dynamic range is usually set by the storage means, file or system 

10 and is typically 0 - 255. In photography, the trick to success is to use all the 

11 dynamic range without exceeding it. In a photo, the trick is to capture both the 

12 details in bright areas and details in dark areas without a loss of details anywhere. 

13 In film photography, when a significant area of the negative has turned 

14 completely opaque it means that the light was so intense in that area that no film 

15 crystals were left unchanged. The variations within the washed out area of the 

16 photograph are lost and can be said to 'have exceeded the dynamic range of the 

17 system'. 

18 In digital photography, the strength of the light energy is measured by 

19 the photo-electric sensors. These values are stored as a true color format computer 

20 file. The dynamic range of the system is exceeded when areas of the grid have 

21 been set to the maximum (say 255). Accordingly, variations within the bright areas, 
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1 hypothetically 256 - 300 can only be recorded as 255 and thus details within such 

2 areas are lost. 

3 With a limited dynamic range, as is the case with digital images, the 

4 amount of light which is captured can significantly affect the image. Consider if one 

5 image is obtained containing one particular dot of light which is measured within the 

6 dynamic range of the recording device. For example, the light may come from a 

7 brown surface. The light dot is measured in terms of three color strengths, the red 

8 (R), green (G) and blue (B). If a second image is obtained having had double the 

9 exposure time, then twice as much light will go into the recording device for each 

10 and every dot, including the dot we are considering. If the range of light remains 

1 1 within the dynamic range of the system, then all the three values of R, G, and B will 

12 be doubled - yet the color of the original brown dot remains as brown. Doubling of 

13 the incoming light means its strength measurement will be doubled - R Is doubled, 

14 G is doubled and B is doubled. 

15 In the case of digital information, for example, upon a doubling of the 

16 light (through a larger aperture or longer exposure), relative RGB values are also 

17 doubled. RGB values of 50,30,20 are doubled to 100, 60,40, because 100 = 50*2, 

18 60 = 30*2, and 40 =20*2. 

19 When twice as much energy hits each sensor, then each sensor has 

20 twice the stimulation. Twice as much red energy hits the red sensor and the other 

21 energy doesn't matter to it. Twice as much green energy hits the green sensor. 

22 And twice as much blue energy hits the blue sensor. 
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1 As the light gathering power Increases, the three measurements of the 

2 primary colors of the same dot increase proportionally as shown in the following 

3 Table 1 - hence the color remains the same, only brighter. 

4 Table 1. - Effect of Increasing Light Gathering Power on Measurements of RGB 



5 
6 
7 


Light 
Gathering 
Power 


Description 


Red 


Green 


Blue 


Green/Red 


Blue/Red 


8 


1 


"Reference, one unit" 


50 


30 


20 


0.600 


0.400 


9 


1.1 


Up 10% 


55 


33 


22 


0.600 


0.400 


10 


1.2 


Up 20% 


60 


36 


24 


0.600 


0.400 


11 


1.3 


Up 30% 


65 


39 


26 


0.600 


0.400 


12 


1.4 


Up 40% 


70 


42 


28 


0.600 


0.400 


13 


1.5 


Up 50% 


75 


45 


30 


0.600 


0.400 


14 


2 


Double 


100 


60 


40 


0.600 


0.400 


15 


2.5 


Two and a half times 


125 


75 


50 


0.600 


0.400 


16 


3 


Triple 


150 


90 


60 


0.600 


0.400 


17 


4 


Four times 


200 


120 


80 


0.600 


0.400 


18 


5 


Five times 


250 


150 


100 


0.600 


0.400 


19 
















20 




In Table 1 , note that the ratio of Green/Red and the ratio of Blue/I 


21 


remain constant, regardless of the light gathering power. 







22 The reference level for light gathering power is artificial, a matter of 

23 convenience. What is the 'correct' measurement of the color of the dot of Table 1 ? 

24 An image collected from an overcast outdoors environment may measure the color 

25 dot at 50,30,20 and the color dot measured in a bright indoor setting could be 

26 200,1 20,80 utilizing four times the light gathering power. 
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1 While it is possible to calibrate light gathering power in terms of the 

2 energy that is collected, it is rarely done in practice. The light gathering power of 

3 our own eyes naturally varies. City lights that seem so very bright at night appear 

4 dull in daylight because the eye's iris automatically opens at night and then closes 

5 during the day, as is required to deal with the natural extreme variations in light 

6 level. Applicant is not aware of a recording device that has a wide enough dynamic 

7 range to handle the range from natural daylight to artificial city night lighting with the 

8 same light gathering settings. What makes a good image is, in part, that the light 

9 gathering ability is varied with the light - so that the recording is not pushed outside 

10 of the dynamic range. Both measurements of 50.30,20 and 200,120,80 for the 

1 1 same color or dot are valid. 



12 For any set of three numbers, such as the intensity values for each of 

13 Red, Green, and Blue, there are a total of six possible ratios and they are G/R, B/R, 

14 G/B, R/G, R/B, and B/G. Only two of the ratios are unique, the other four ratios are 

1 5 redundant as they are variations of the first two. 

16 For example, based upon G/R and B/R, the others are: 

17 green/blue = (green/red) / (blue/red) 

18 red/green = 1 / (green/red) 

19 red/blue = 1 / (blue/red) 

20 blue/green = (blue/red) / (green/red) 

21 In principle, any two of the six can be chosen. For the purposes of 

22 this description, the ratios of green/red and blue/red are chosen. 
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1 An efficient choice for the value of the strength of the RGB triplet 

2 would be to simply take the maximum value from amongst the three RGB values. In 

3 the case of 50,30,20 for RGB respectively, red happens to be the maximum value 

4 and the calculation of strength and the two ratios becomes: 

5 Strength = red = 50 

6 Ratiol = green/red = 30/50 = 0.600 

7 Ratio2 = blue/red = 20/50 = 0.400 

8 In reverse, one can back-calculate and recover the red, green and 

9 blue values as follows: 

10 red = Strength = 50 

1 1 green = Strength * Ratiol = red * green / red = 50 * 0.600 = 30 

12 blue = Strength * Ratio2 = red * blue / red = 50 * 0.400 = 20 

13 This artificial representation of the RGB triplet is useful because, for 

14 any dot, the two ratios are independent of the light gathering power. The amount of 

1 5 light gathered only affects the strength component. 

16 Too much light gathering will cause the measurement of the color to 

17 become distorted because at least one of the three primary colors RGB will exceed 

18 the dynamic range and thus will not be accurately represented. Consider the 

19 previous example dot of 50, 30, 20 at some arbitrary reference level of light 

20 gathering power. If the image is re-recorded, but at a much higher light gathering 

21 power, then at least one color will be pushed beyond the dynamic range. 
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Table 2. - Increasing Light Gathering Power beyond the Dynamic Range of System 



2 
3 
4 


Light 
Gathering 
Power 


OescriDtion 


Red 


Green 


Blue 


Green/Red 


Blue/Red 


5 


1 


Reference 


50 


30 


20 


0.600 


0.400 


6 


3 


Triple 


150 


90 


60 


0.600 


0.400 


7 


5 


Five times 


250 


150 


100 


0.600 


0.400 


8 


5.1 


5.1 times 


255 


153 


102 


0.600 


0.400 


9 


5.2 


5.2 times 


255 


156 


104 


0.612 


0.408 


10 


6 


Six times 


255 


180 


120 


0.706 


0.471 


11 


7 


Seven times 


255 


210 


140 


0.824 


0.549 


12 


8 


Eight times 


255 


240 


160 


0.941 


0.627 


13 


9 


Nine times 


255 


255 


180 


1.000 


0.706 


14 


10 


Ten times 


255 


255 


200 


1.000 


0.784 


15 


11 


Eleven times 


255 


255 


220 


1.000 


0.863 


16 


12 


Twelve times 


255 


255 


240 


1.000 


0.941 


17 


13 


Thirteen times 


255 


255 


255 


1.000 


1.000 


18 


1 


"Reference, one unit" 


50 


30 


20 


0.600 


0.400 



19 Notice that neither the ratio of green/red nor blue/red are constant 

20 (because they have saturated) past 5.1 times the light gathering power. Because 

21 there is a maximum amount of energy that can be measured by the recording 

22 device (either digital sensors or film) there is a practical limit to the useful light 

23 gathering power for any subject. In Table 2, when light gathering power exceeds 

24 five times the reference power, the dynamic range is almost fully used. The 

25 strength of the dot is 250 and the maximum that can be stored Is 255. That 

26 maximum is reached, exactly, at 5.1 times the light gathering power. At 5.1 the 

27 strength of the dot is 255 yet the two ratios are still 0.600 and 0.400 respectively. At 

28 5.2 times the light gathering power, even more light enters the recording device. 
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1 The red sensor would have stored the number 260 but it cannot because the 

2 dynamic range is surpassed and the value is clipped to 255 instead. The correct 

3 numbers are stored for both the green and blue sensors, however, the incorrect 

4 total color is recorded and this is revealed by noticing that the two ratios now vary 

5 from the proper ratios of 0.600 and 0.400. At 6 times the light gathering power, 

6 even more light enters and the distortions are correspondingly greater. The ratios 

7 are now significantly different and incorrect at 0.706 and 0.471 and these 

8 correspond to a significantly different color. At 8.5 times the light gathering power, 

9 the blue sensor also measures clipped values. The light is recorded as having the 

10 same strength in both the red and blue primary colors. The blue to red ratio is now 

11 1 .000 and the previous distorted color (reddish brown) now further distorts to an 

12 orange. Finally, at 12.25 times the light gathering power, all three sensors clip and 

13 the color is recorded as three full strength primary colors, meaning white. 

14 
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1 It is important to avoid collecting light beyond the ability of the 

2 recording system (it's dynamic range) because it causes distortion in the color. 



3 Any one of the RGB triplet can be chosen as the reference color. If 

4 green was the strongest color, and assuming the color was 30,50,20 (having the 

5 same reference light gathering power as the previous red example), then having 

6 reference to Table 3, the same behavior is exhibited. 

7 Table 3. - Effect of Increasing Light Gathering Power with Green as strongest color 



8 Light 

9 Gathering 



10 


Power 


Description 


Red 


Green 


Blue 


Green/Red 


Blue/Red 


11 


1 


Reference 


30 


50 


20 


1.667 


0.667 


12 


3 


Triple 


90 


150 


60 


1.667 


0.667 


13 


5 


Five times 


150 


250 


100 


1.667 


0.667 


14 


5.1 


5.1 times 


153 


255 


102 


1.667 


0.667 


15 


5.2 


5.2 times 


156 


255 


104 


1.635 


0.667 


16 


6 


Six times 


180 


255 


120 


1.417 


0.667 


17 


7 


Seven times 


210 


255 


140 


1.214 


0.667 


18 


8 


Eight times 


240 


255 


160 


1.063 


0.667 


19 


9 


Nine times 


255 


255 


180 


1.000 


0.706 


20 


10 


Ten times 


255 


255 


200 


1.000 


0.784 


21 


11 


Eleven times 


255 


255 


220 


1.000 


0.863 


22 


12 


Twelve times 


255 


255 


240 


1.000 


0.941 


23 


13 


Thirteen times 


255 


255 


255 


1.000 


1.000 



24 As shown, the same situation occurs wherein the green/red and 

25 blue/red ratios become distorted in a veiy similar manner. In this case, the ratios 

26 red/green and blue/green produce the same numbers (0.600 and 0.400) as in the 

27 previous 50,30,20 illustration for the case of red. 
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1 Accordingly, regardless of which of the RGB triplet is the maximum, 

2 there is a symmetry and strength is defined as the maximum strength value of (Red, 

3 Green, and Blue) and color ratios result (red/Strength, green/Strength, 

4 blue/Strength). One of the three ratios will be exactly equal to 1 because strength Is 

5 always equal to one of the numerators. In the case of Red from Table 1 , RGB = 

6 50,30,20 so that strength = max of (50,30,20) which is 50 and the color ratios are 

7 (50/50, 30/50. 20/50) = (1 . 0.600, 0.400). 

8 Thus, for Red, Green and Blue respectively: 



9 


Color 


Strenath 


R/S 


G/S 


B/S 


10 


50.30.20 


Red 


1 


0.6 


0.4 


11 


30.50.20 


Green 


0.6 


1 


0.4 


12 


30.20,50 


Blue 


0.6 


0.4 


1 



13 As considered before, when a camera gathers more light compared to 

14 another setting for the same image, a given dot has the qualities that the ratios of 

15 the measured primary colors remain the same - if we are within the dynamic range 

16 of the recording system. Instead of thinking of the dot as an RGB triplet we can 

17 think of the dot as a strength and ratios. Again, for our dot, 50,30,20 at its reference 

18 strength of 50, the color ratios are 1 , 0.600 and 0.400. 

19 

20 
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1 Enhancing the Image 

2 As long as the color ratios are unaltered, the image can be adjusted 

3 without adversely affecting the colors. For instance, should insufficient light have 

4 been gathered by the recording device, we can virtually amplify the light power or 

5 strength while maintaining the color. 

6 To virtually amplify or scale the light by a correction factor of 2, the 



7 strength is doubled without varying the color ratios: For example, doubling the 

8 strength of our dot of (50,30,20) to 1 00 results in a color dot of 50 * 2 * (1 ,0.6,0.4) = 

9 (100,60,40). Simply, the same result can be achieved by simply multiplying the 

10 R,G, andBby2. 

1 1 This virtual true color light amplification is accomplished by multiplying 

12 or scaling all three primary colors by the same number. No colors become 

13 distorted, as long as the output values stay within the dynamic range. The dynamic 

14 range is exceeded if a R, G, or B value is calculated that is greater than the range 

15 used by the file format for the image (such as 255). 



16 One can ensure that no value, resulting from the triple multiplication 

17 scaling, can exceed the dynamic range, by deriving the correction from an algebraic 

18 scaling expression or function. 

19 Having reference to Fig. 1, the X-axis represents the strength of an 

20 image dot (Maximum of red, green, and blue). The scales of 0 - 1 represent the 

21 limits of the dynamic range (such as 0 - 255). For the linear diagonal shown, a 



23 



1 strength of 50 (50/256 - 0.2) for a dot is scaled by unity for an output value of 0.2. 

2 Accordingly, the unity diagonal corresponds to a 'no operation' situation where the 

3 output is Identical to the input. However, once the scaling function deviates from 

4 unity, the output values will be different from the input values, resulting in a change 

5 to the image. 

6 To scale the dynamic range from 0 to 1 .0 Is to simply divide a cun'ent 

7 strength (maximum of the RGB triplet) value by the maximum number that can be 

8 stored with this dynamic range. Suppose that number is 255. An arbitrary dot will 

9 have a dot maximum strength having a number between 0 and 255. To scale the 

1 0 dot maximum to 0 and 1 .0 one divides the strength by 255. 

11 Both the input and the output axis represent the values of the 

12 maximum of the RGB triplet; the dot maximum. The input is the dot maximum 

13 under consideration. The output con^esponds to the adjusted dot maximum for the 

14 RGB triplet that will be calculated as a result of the method. 

15 

16 Dot Maximum 

17 One method to find the maximum of an input RGB triplet is to choose 

18 one as maximum, testing each of the others, and resetting the maximum to that 

19 other if higher. 

20 
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1 The following 3 lines of pseudo code illustrated the selection of a dot 

2 maximum: 

3 strength = red 

4 strength is set to red value 

5 IF strength < green THEN strength = green 

6 if strength is less than green value then reset it to the green value 

7 IF strength < blue THEN strength = blue 

8 if strength is less than blue value then reset it to the blue value 

9 Having reference to Fig. 1 , any scaling or correction is constrained to 

10 the 1 X 1 graph shown. The input axis is constrained to the domain from 0 to 1 and 

1 1 the output axis is constrained to the range of 0 to 1 . This means that the strength of 

12 an adjusted or corrected dot will not exceed the dynamic range. 

13 Any scaling function that can be plotted within the constrained graph 



14 can be used for virtual true color light amplification. The properties of a particular 

15 graph will affect the final aesthetics and application. A particular function is chosen 

16 as appropriate for the application; whether it be to adjust the brightness of an entire 

1 7 image, or a portion of the image, or other adjustment. 

18 Two implementations of the scaling function correction include forming 

19 a lookup table of corrections (a finite number dictated by the dynamic range); and 

20 another less efficient means is to calculate each dot independently in turn. One can 

21 understand that in an image, the value of the strength of a particular dot may be 

22 repeated many times for many other dots. Thus, for efficient calculation, in the case 

23 of a look-up table, a correction can be calculated only once but applied many times. 
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1 For the look-up table approach, once the parameters needed to 

2 specify a particular scaling function are known - all the corrections (usually 256 of 

3 them, this depends on the limit of the dynamic range) are then calculated by means 

4 of a subroutine and the results stored in a look up table so that each is only 

5 calculated once. The correction is then simply looked up in this way (a pseudo 

6 coded example is:) 



7 corr = corra(strength) 

8 where corr is the particular correction for the current dot; 

9 corraO is the lool<up table or array that stores the corrections; and 

1 0 strength is the dot maximum of the RGB for the current dot. 

11 The other (less efficient) way is to have a function calculate the 

12 correction for a particular dot, each one in turn. An example would be 

1 3 corr = correct(strength) 

1 4 where corr is the particular correction for the current dot; 

1 5 correct () is the correction function which executed by 

1 6 simply 'calling' its name In this way; and 

1 7 strength is the dot maximum of the RGB for the current dot. 

18 A given point along the graphed scaling function, which provides an 

19 input and output value, is to be used to derive the correction multiplier or factor. A 

20 correction factor is equal to output value / input value. 

21 So as to maintain the respective color ratios, all three of the RGB 

22 triplet values are multiplied by this same correction corr as determined for and 

23 specified by the dot maximum. 
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1 Accordingly, red = red * corr; green = green * corr; blue = blue * corr 

2 where red, green, blue end up holding the values for the current dot before and after 

3 connection. The effect of coupling these four considerations Is that of Virtual True 

4 Color Light Amplification. The Dynamic Range is never exceeded and the color is 

5 always preserved. 

6 

7 Practical Imolementation 

8 An image can be read in various ways. Applicant has avoided the 

9 need to review the various graphical computer file formats by illustrating the method 

10 on a displayed image. Applicant is aware that, currently, Visual Basic (a 

11 programming language operable under the Windows operating system - all 

12 trademarks of Microsoft Corporation) and most other modern programming 

13 languages have simple commands that allow for image reads. In Visual Basic, one 

14 command is pbox.Picture = LoadPicture(fileJn) where pbox is a 'picture box object', 

15 used for displaying pictures, Picture is a 'method' which assigns a picture to the 

16 object, LoadPictureO is the function that reads Picture Files, and filejn is the name 

17 of the file that is to be read. Once this command is issued by Visual Basic, the 

18 image file Is read in and displayed on the screen in the programming 'tool' called the 

19 'picture box object'. There is a similar method to save a picture. 

20 Having reference to Fig. 2, the simplified code illustrates a Visual 

21 Basic implementation of the virtual true color light amplification method applied to 

22 an image. This simplified technique requires, at a minimum, a 16 bit video card and 
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1 a 24 bit card is preferable. The code of Fig. 2 is directed to extracting color values 

2 from the video card itself. This is not the most efficient technique and could be 

3 improved significantly by storing the image in the main RAM memory. This would 

4 eliminate accessing the video card at all and eliminate the extraction steps of 

5 stripping R, G and B values from a combined color variable such as that returned by 

6 Visual Basic function pbox.Point(icoUrow). Accessing memory could result in about 

7 a 7 times efficiency gain. 

8 Simply, the process permits a rectangular displayed image of dots to 

9 be adjusted. For example, the image may be dark, only having a maximum 

10 strength for any of the dot maximums being about 128, or half the dynamic range 

11 for the system. In the simplest case, the image range is scaled to the dynamic 

12 range as a linear function. Accordingly, by normalizing the maximum of 128 to 255, 

13 the strength of all dot maximums will be doubled. Accordingly, the scaling function 

14 is merely a constant of 2 and the look up result, for any dot maximum, is 2. For 

15 each column of the image, the values of the color are extracted for blue, green and 

16 red. The dot maximum is set as red and the green and blue are tested to reset the 

17 strength to the maximum amongst the three. The correction is looked up in the 

18 table, in this case being a constant of 2. Each of the values for RGB are scaled by 

19 2, the maximum scaled value being 128 * 2 or 256 - the maximum of the dynamic 

20 range. The modified dot is written back to the display, all colors having been 

21 preserved and without having exceeding the dynamic range. 

22 
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1 Applications - Effect of the Scaling Function 

2 The virtually infinite selection of scaling functions, as applied to the 

3 image, results in different effects to suit various differing objectives. Sometimes the 

4 quality of the image dictates which function is used (such as a function designed 

5 simply for brightening a dark image), or a more particular function which enhances 

6 only specified strengths within an image (such as extracting detail from a narrow 

7 portion with minimal concern regarding the effect on the other portions of the 

8 image). 

9 

10 Applications - Virtual Lens 

11 Both the aperture and the shutter cause problems in and of 

12 themselves. If the subject of the photograph is moving, the shutter can only be 

13 open for a short period of time or the image will be blurred by the motion itself. 

14 When the aperture is opened up, the depth of field (which means the 

15 range of distance that is in focus) decreases. Even when focusing correctly, a wide 

16 open aperture means that only a small range of distance will be in focus. This 

17 'distortion' is due to the spherical shape of the lens itself. When the aperture is 

18 small, the depth of field is better because only the nearly flat center of the lens was 

19 used. The smallest aperture setting provides the largest depth of field. 

20 In practice, photography (and light gathering in general) is a trade off 

21 of these two effects. A motionless scene can have a large depth of field, by 
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1 choosing a small aperture and a slow shutter speed. A racing car can only be 

2 photographed at the cost of the depth of field, the shutter cannot be open for long 

3 and so the aperture must be opened to gather more light. 

4 Suppose that a 'normal' quality lens is used to photograph something 

5 that is moving quickly but the photographer does not want to lose the depth of field. 

6 Without the process of the present invention there is no way to do this. Accordingly, 

7 by setting the controls to gather too little light, in terms of normal photographic 

8 thinking, the uncorrected image will be too dark but the depth of field is preserved. 

9 Suppose the photographer had collected a quarter (25%) of the amount of light that 

10 would make full use of the dynamic range. The image will be nearly black, with the 

11 highest value recorded being 63 whereas the dynamic range limit is 255. This 

12 image can be corrected to bring out the detail in the dark areas. 

13 Accordingly, in a first embodiment, and having reference to Fig. 3, the 

14 graph is a straight line which terminates at the point (x, 1.0) where x is the 

15 maximum of the entire measured image. This maximum value can be found using a 

16 modified histogram approach. In this example, x would be 0.25 but it could be any 

17 value between 0 and 1 . In the preferred embodiment of the invention, virtual light is 

18 added in the same way that opening the aperture more would have except that it 

19 will have a depth of field associated with a superior lens. This process can also be 

20 used to make up for 'blunders' where inappropriate lens settings resulted in a too 

21 dark photograph. 

22 
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1 Virtual Iris 

2 In a second embodiment, and having reference to Fig. 8. a graph can 

3 be chosen so that the darkest parts of the image will remain dark, the duller parts 

4 will brighten but also so that the bright parts of the image remain nearly unaffected. 

5 Fig. 8, and similarly shaped smooth non-linear graphs, have the effect of imitating 

6 the iris when used with the virtual true color light amplification of the present 

7 invention. The output image happens to more closely resemble one's visual 

8 memory of the experience. Applicant refers to this enhancement as "virtual iris". 

9 The gentle nature of this non-linear graph ensures that a quality input 

10 image will result in an attractive processed image. The important aspects to 

11 maintain this aesthetic result is that the graph remains smooth, the slope of the 

12 graph is never zero and is also smoothly changing, and that there is a net 

1 3 brightening effect in total. 

14 In Fig. 8, the scaling function approaches the asymptote of the 

15 minimum and maximum of the system's dynamic range. The more the function 

16 approaches a tangent to the minimum and maximum of the dynamic range, the 

1 7 more severe the correction. 

18 

19 Virtual Detail Enhancement 

20 Simply, any one frame of a photo is the result of only one aperture and 

21 shutter setting. In investigative work, this has the annoying limitation that details in 
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1 certain areas of the photograph will be subtle, in a third embodiment, a process is 

2 provided for bringing out detail in that certain subtle area. 

3 Having reference to Fig. 4, an area of interest is chosen and a 

4 histogram approach is applied to find that the minimum strength dot within the area 

5 is 0.30 and the maximum strength dot is 0.50. This area utilizes only 20% of the 

6 dynamic range, which means that the contrasts will be subtle - or virtually 

7 indistinguishable to the eye. 



8 Applying a three part linear scaling function as shown in Fig. 4 turns 

9 these small contrasts into large ones as the output from that specified area now 

1 0 varies over 80% of the dynamic range. 

1 1 The small sloped lines below 0.30 and above 0.50 on the input have 

12 the effect of washing out the detail in darker and brighter parts of the image. But 

13 the resulting color, at each dot, will never be corrupted and the brighter parts of the 

14 image provide a good reference. 

15 Any area of the photograph can be chosen. Having reference to 



16 Fig. 5, details in a very dark area are revealed, such as writing obscured in shadow. 

17 Fig. 6 illustrates how to bring out the details in a bright area, such as tracks in the 

18 snow. Any number of areas can have their detail enhanced by simply choosing the 

19 area of interest and applying the correction. 

20 More particularly, any areas or portions of the image can be 

21 optimized. First, the area needs to be identified. In a Graphics User Interface, this 
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1 is easily done using the mouse in a 'click and drag' operation. This can be done in 

2 Object Oriented Programming by using the Operating System (Windows) to identify 

3 when the mouse button has been elicited. In Visual Basic there are built in 

4 subroutines (for every program) that are executed as soon as the mouse button is 

5 depressed or released. 

6 A user can select any rectangular area within the image. The 

7 'coordinates' are stored in common memory as xdwn, ydwn, xup and yup. See the 

8 photographic examples #1 and #2 for the superimposed rectangle on the image. 

9 In both the Virtual Lens and the Virtual Detail Enhancement, I referred 

10 to a 'modified histogram approach'. A histogram is simply the measurement of the 

1 1 number of occurrences against the value of the occurrences. 

12 
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1 



For example, as shown in Table 4, In terms of a set of RGB triplets: 



2 Table 4 - Standard Histogram Approach 

3 "Red, Green, and Blue values taken as independent" 

4 Dot# Red Green Blue Value # Events 

5 1111 

6 2 0 0 1 0 2 

7 3 2 2 1 1 11 

8 4 4 3 1 2 3 

9 5 4 4 4 3 7 

10 6 3 2 1 4 5 

11 7 5 11 5 2 

12 8 3 3 1 

13 9 4 3 1 Total 30 

14 10 3 3 5 

15 In this example, there are 10 dots each having 3 values (red, green 



16 and blue) and each of these 30 values range in value from 0 to 5. The normal 

17 histogram is calculated by adding up the number of times each value (0, 1, 2, 3, 4, 

1 8 and 5) occurs in total. 

19 
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1 The modified histogram approach, where strength = max(red, green, 

2 blue), described by Table 5 as follows: 



3 Table 5 - Modified Histogram Approach 

4 "Red, Green, and Blue values taken as a unit, Histogram on maximum" 

5 Dot# Red Green Blue Strenqtii Value # Events 

6 1111 1 

7 2 0 0 1 1 0 0 

8 3 2 2 1 2 1 2 

9 4 4 3 1 4 2 1 

10 5444 432 

11 6 3 2 1 3 4 3 

12 7 5 1 1 5 5 2 

13 8 3 3 1 3 

14 9 4 3 1 4 Total 10 

15 10 3 3 5 5 

16 Simply, it is the occurrence of numbers as found in the maximum 

17 strength that is used to build the histogram and not the values of red, green and 

18 blue separately. 

19 This is in keeping with the nature of this patent application which is 

20 that red, green, and blue values are to be treated as a unit having a strength and 

21 ratios and not as three independent values. 

22 The histogram is built by considering only those dots within the range 

23 of rows = xdwn to xup and columns = ydwn to yup. 

24 Having reference to Fig. 7a, example code is provided by which to 

25 apply the modified histogram. 
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1 At this point, the histogram is formed and its running total is known 

2 with respect to the strength of the RGB triplets of the marked area. The beginning 

3 and the ending significant strengths are determined, as reflected by the histogram 

4 data. 

5 To avoid errors such as dead or saturated recording elements and 

6 othenA^ise 'stray values' that are not representative of the area, one can limit the 

7 relevant dots to the 2% and the 98% of the number of occurrences to represent the 

8 smallest and largest relevant RGB strengths. 

9 The number of strengths counted by the histogram, in total, is equal to 

10 the last running total value, and the 2% and 98% values are, therefore, easily found. 

1 1 Code is shown in Fig. 7b which determines the range of strength index (hmin and 

12 hmax) corresponding to the range of strengths within the box selected by the user. 

13 Having reference also to Fig. 4, the modified histogram approach 

14 found 0.30 (of the dynamic range maximum) and 0.50 (of the dynamic range 

15 maximum) to be the minimum and maximum strength values of the portion of the 

16 image selected by the user. (See photographic examples #1 , #2, for the boxes). 

17 it was also assumed that the range calculated from the modified 

18 histogram approach should be modified so that it varied over the majority of the 

19 dynamic range. 

20 So, what should this input range of hmin and hmax be turned into? 

21 We want it to occupy most of the dynamic range. A good guess is 80% of the 



36 



1 dynamic range with a little left over for the darker and lighter areas so they can still 

2 be used as a reference. 

3 In application code, the output strength range of the selected area was 

4 originally set to 0.1 to 0.9 of the dynamic range maximum. It was later reset to 0.2 

5 to 0.9 as these numbers simply seemed better after observing many images. 

6 The code used to calculate the look up table based on the modified 

7 histogram approach for determining the input strength range and the (nearly) 

8 arbitrary output range of 0.2 to 0.9 follows. 

9 The graph of Fig. 4 can be thought of, in the general sense, as having 

10 3 line segments each with two end points. 

1 1 Table 6: Line Segments: 

12 Line Segment Start Stop 

13 #1 (0,0) (xmin,0.20) 

14 #2 (xmin,0.20) (xmax,0.90) 

15 #3 (xmax.0.90) (1.0.1.0) 

1 6 where xmin and xmax have been calculated by the modified histogram 

17 approach. That is to say: 

18 xmin = hmin / drmx 

1 9 xmax = hmax / drmx 

20 Both input axes are measured in terms of the dynamic range. The 

21 input values are in terms of the strength (max of RGB) of the dot. The graph never 
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1 leaves the 0 to 1 'box'. These constraints must always be met by any scaling 

2 function in any specific process. 

3 All that remains, here, is that the output value be calculated by a 

4 program equivalent to that described for Fig. 2 above and that the lookup table does 

5 not hold the graph, exactly, but the ratio of the output to the input. 

6 Each line segment can be expressed with algebra in the 'slope 

7 intercept' form, of which the general form is: y = m*x + b. For each of the three line 

8 segments, linear equations and scaling factors are determined. An array of 

9 corrections or scaling factors can be formed from the three equations. Dividing the 

10 output values by the system dynamic range produces the ratio of output to input. 

1 1 This virtual detail enhancement technique, or forensic flash due to its 

12 ability to delve into the normally obscured areas, maximizes the dynamic range of 

13 any target so that the details are enhanced. This is not restricted to the target area 

14 but any part of the photograph that has similar strengths to the user's choice will 

15 also be so enhanced. Any target area can be selected and so there can be many 

16 valuable corrections performed on the same photograph. Those areas which were 

17 stronger than the strength range picked by the user remain as useful references 

1 8 due to the 'true color" nature of the correction. 

19 
20 
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1 Examples 

2 Virtual Flash - Virtual Iris 

3 The examples illustrated in Figs. 9 and 10 illustrate corrections to the 

4 limitations that occur from the physical light gathering devices or image recorders. 

5 The one aperture and shutter setting per frame means that a photograph is likely to 

6 vary from one's memory of the experience of being there. The eye's iris adjusts 

7 itself when experiencing contrasts. In a park on a sunny day, the iris opens up 

8 when moving from sunlight to the shade so that you remember all the grass as 

9 being green whereas photos often show shaded grass as black. 

10 

11 Photo #1 (Flos. 9a - 9f) Tourist photo of an Abbey 

12 This example illustrates how various prior art processes and the 

13 present invention enhance the image. The approach . with the prior art processes is 

14 to "play" with the image until the brightness looks about right for what you want. It is 

15 a subjective thing and the expert user is someone who is good at making the 

16 necessary compromises. In Fig. 9a, the original image, scanned from a 

17 photograph, is very dark but still uses all/most of dynamic range (of the dot 

1 8 maximums (hits) being outside the strengths of 5 and 254) of a system of 256. 

19 Figs. 9b - 9d illustrate prior art image brightening techniques. Fig. 9b 

20 does so by increasing the image brightness by 80%. While the image of Fig. 9b is 

21 brighter, the colors are badly faded and the sky has also experienced change in 
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1 color. Fig. 9c illustrates the prior art brightened image of Fig. 9b with contrast set to 

2 50%. Contrast is increased in an attempt to try to restore the colors lost in 

3 brightening. Notice how much of the detail of the image is lost. The process has 

4 pushed many of the dots past the edge - outside of the dynamic range. Fig. 9d 

5 illustrates the brightened image of Fig. 9b with the saturation set to 50%. Increase 

6 in saturation is another technique for restoring the colors. As a result, the sky is 

7 almost returned to what it was but the rest of the image has significant and unsightly 

8 color distortions. 

9 Applying the techniques of the present invention, the photo fairs much 

10 better particularly in Fig. 9f. In Fig. 9e, the image range of 5 to 254 is linearly 

1 1 mapped to 0 to 255. The effect is small because the original image was nearly full 

12 range already. It does, however, ensure that we have the full dynamic range in the 

13 output image. In Fig. 9f, the scaling function of Fig. 8 was applied for obtaining a 

14 superior image. All the colors are true to the image, as it was scanned, and are 

1 5 vibrant, just as they would have been to the eye with no loss of detail. 

16 

17 Photo #2 (Fias. 10a - lOf) Tourist ohoto of Stone Henae 

18 In the original frame of Fig. 10a, the stones are in the shadows due to 

19 the extreme lighting conditions. In Fig. lOf, the virtual iris process of the present 

20 invention compensates in a similar way that the iris does automatically, turning the 

21 poor photo into a good one. 
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1 More particularly, in Fig. 10a, the subject is very dark and again uses 

2 all/most of dynamic range (only 1% of the hits being outside of a strength of 6 and 

3 253). This image had been "pre-processed" by others to bring out detail - notice 

4 how the sky is nearly white but clouds are still available (reproduction of the Figures 

5 in this application does not necessarily preserve the actual presence of the clouds). 

6 The prior art had taken it "as far" as its could but the subject was still too dark. Fig. 

7 10b illustrates prior art brightening of the image by 60%. While image is brighter, 

8 the colors are badly faded and the stones have lost all their color. Some of the 

9 detail is also lost by this process alone. Note, even in the gray-scale rendering, the 

10 whitening of the "red" rock left of the stones and at the left extreme of Fig. 10b 

1 1 image. Fig. 10c is the brightened image of Fig. 10b with contrast set to 40%. Notice 

12 that the stones have, in areas, regained some color but not in other areas. Also 

13 notice how much more of the detail of the image is lost. Fig. 10d is the brightened 

14 image of Fig. 10b with saturation set to 15%. There is improved color that is "sort of 

15 right in a way" but it also adds artificial colors, such as some reds and yellows. 

16 Even in photos where not much correction is needed, manipulation of saturation for 

17 each dot ends up with different RGB ratios than were recorded. At best, one ends 

18 up with a compromise solution. 

1 9 Applying the techniques of the present invention. In Fig. 1 0e the image 

20 range of 6 to 253 is linearly mapped to 0 to 255 to ensure the full dynamic range in 

21 the output image, in Fig. lOf, the scaling function of Fig. 8 was again applied for 

22 obtaining the superior, true color image with no loss of detail. 
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1 

2 Forensic Flash - Virtual Detail Enhancement 

3 Photo #3 (Fias. 1 1 a. 11 b) Satellite 

4 In Fig. 11a, the satellite is in the shadows and the surface is very dark. 

5 This often happens in space because of the extreme contrast in lighting. Important 

6 'docking' holes cannot be seen. Using the forensic flash correction technique of the 

7 third embodiment, a window or box was selected within the dark area and the 

8 histogram approach used to build a correction graph suited for that area. 

9 In Fig, 11b, the processed image now shows the detail in the dark 
10 area. The docking holes, marked with white circles, are now revealed. 

11 

12 Photo #4 (Fig. 12a. 12b) Blimp 

13 In the original image of Fig. 12a, the underside of the blimp is in the 

14 shadows. This often happens in photography when the lens is directed into the sky. 

15 Specifically, the identification markings cannot be seen as the tail is too dark. 

16 Under the principles of the third embodiment, the tail area is selected 

17 and the histogram approach used to build a correction graph suited for that area. 

18 As a result, as shown in Fig. 12b, the processed image shows the lettering in the 

19 previously obscured, dark area. The blimp is now identified as COLUMBIA N3A. 

20 
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1 Photo #5 (Fig. 13a. 13b) Car Plate 

2 In the original frame of Fig. 13a, the car's license plate is mostly 

3 shrouded in shadows. The car cannot be identified because the plate cannot be 

4 read. Using the third embodiment, the plate is selected and the histogram approach 

5 used to build a correction graph suited for that area. 

6 As a result, and refening to Fig. 13b, the processed image shows that 



7 the car does not have a normal plate at all but, instead, the words: Classic Mustang. 
8 

9 Photo #6 (Fig. 14a. 14b. 14c) Tracks in the Snow 



10 In Fig. 14a, there are two people skiing. Their tracks are identifiable, 

1 1 but subtle. Using the forensic flash con-ection technique of the third embodiment, a 

12 window or box was selected within the overexposed area of the tracks in the snow 

13 and the histogram approach used to build a con-ection graph suited for that area. 

14 In Fig. 14b, the processed image now shows the detail in the 

15 overexposed area. The tracks in the snow are distinctly visible. 

16 Similarly, a window or box was selected in the dark area of the face of 



17 the skier in the photograph. Figure 14c Is the result of the histogram approach used 

18 to build a correction graph uniquely suited to this area of the photo. The features of 

19 the skier are more clearly visible than in the original photo shown in Fig. 14a. 

20 
21 
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1 Summary 

2 The key concepts are here expressed as the combination of the 

3 following six factors: correcting in RGB color space, the correction graph; the 

4 definition of the correction axes; constraint of domain and range to the system's 

5 dynamic range; properties of the graph; and application of the same correction 

6 factor to each of R,G and B in the triplet. 

7 The correction must be applied to the RGB color space to maintain 

8 true color. Any correction graph can be used that embodies the above 

9 characteristics. Both the input and the output axes represent the maximum of the 

10 RGB triplet. The input is the maximum of the RGB triplet under consideration and 

1 1 the output corresponds to the maximum of the RGB triplet that is calculated as a 

12 result of virtual true color light amplification. The correction is constrained to the 

13 dynamic range. This means that the strength of the calculated dot is constrained to 

14 the dynamic range. Any graph that can be plotted within the constraints can be 

15 used for the process. The properties of a particular graph will affect the emphasis 

16 of the correction. Again, all three of the RGB values must be multiplied by the 

17 scaling factor derived from the graph. A given point on the graph has an input and 

18 output value. The correction equals the ratio (division) of these two and all three of 

19 the RGB triplet values are multiplied by this ratio of output to input. 

20 The result of these considerations is a suite of processes all of which 

21 preserve the essential color of each and every dot in the input digital image while 

22 varying the effective light gathering power which can easily be on a dot to dot basis. 
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1 All that differs between the above embodiments Is to vary the graph 

2 within the imposed constraints. With virtual true color light amplification, a new and 

3 useful result is dependent only upon identifying an image enhancing need and 

4 identifying a reasonable graph to fit that need. 

5 

6 CORRECTION OF FADED IMAGES 

7 In a preferred aspect of the invention there is an additional automatic 

8 correction which can be applied to supplied images which are degraded, those that 

9 look faded but lack the natural chaos in the degradation. Through investigation, it 

10 appears to applicant that this is a common problem of images. The problem is that 

1 1 there appears to be a constant triplet of gray added to these classes of photograph. 

12 Grays are represented by a triplet where R, G, and B at any strength have the same 

13 value. Examples of grays are: 



14 Bright White (255,255,255) 

15 White (200,200,200) 

16 Light Gray (150,150,150) 

17 Gray (100,100,100) 

18 Dark Gray (50,50,50) 

19 Black (0,0,0) 

20 Images, such as a red beach ball, might be represented by a triplet 



21 (150, 50, 50) and thus look pink instead of being truly red at (100, 0, 0). The Virtual 

22 True Color Light Amplification process described above and in co-pending 

23 application 09/548,755 cannot correct these photos as it assumes that the input 
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1 (150, 50, 50) is the right color to begin with. A non-faded photograph is likely to 

2 contain a representation of vibrant color, being that one or more of R, G, or B is 

3 zero. Some examples are: 

4 Bright Red (255, 0, 0) 

5 Bright Yellow (255, 255, 0) 

6 Bright Blue (0, 0, 255) 

7 Dark Red (63. 0, 0) 

8 Dark Yellow (63. 63, 0) 

9 Dark Blue (0. 0, 63) 

10 Mathematically, the existence of vibrant color means that the smallest 

11 strength values for R, G or B encountered in the image should be zero, or near 

12 zero. In faded images, the smallest values are non-zero which applicant suggests 

13 are artificial and represent the addition of gray throughput. 

14 In order to remove this artificial addition of gray from the image the 



15 assumption is made that the smallest numbers encountered In the image would 

16 actually have been zeros if there had been no degradation or distortion. An offset 

17 value, x, is determined so as to reduce the gray to zero. This offset (x, x, x) is 

18 subtracted from each of R, G and B. 

19 With reference to Figs. 18a and 18b, to automatically determine the 

20 value of the offset x, a first histogram of all the independent values of R, G and B is 

21 made of the image. To reject outliers, a majority of the population of color dots is 

22 retained by setting the offset x at a clipping threshold, rejecting 0.1% - 0.5% of the 

23 population of strength values which is equivalent to retaining 99.9 to 99.5 % of the 
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1 values. The difference between the strength value at the clipping threshold and the 

2 minimum of the dynamic range (typically zero) is selected as the offset value x. As 

3 shown in Fig. 19b, the removal of (x, x, x) from each R, G and B (R-x,G-x,B-x) shifts 

4 the histogram to the left which is also the minimum strength of the dynamic range. 

5 Essentially offset x is shifted to zero which has the result of making the supplied 

6 image appear somewhat darker. Basically for faded images, there is an advantage 

7 in first darkening an image before seeking to brighten or adjust the supplied image. 

8 

9 AUTOMATIC STRENGTH SELECTION 

10 If the supplied image is not faded or otherwise contain an 

1 1 inappropriate amount of gray, then the supplied image becomes an original image 

12 for brightening. If there is gray in the supplied image, It is preferable to 

13 automatically oftset the strength values of the supplied image to arrive at a new 

14 image, an original image, for brightening. Note that when the offset approach is 

15 applied, it purposefully results in color distortion as it manipulates the strength of 

16 each of R, G and B in different proportions. 

17 The true color brightening or light amplification techniques, 

18 implemented herein, as set forth in the co-pending application and reproduced 

19 substantially above, brighten images based on a function selected by the user or 

20 pre-selected for correction at some pre-determined amount, all without exceeding 

21 the dynamic range and also without color distortion. However, aesthetics are rarely 

22 that simplistic or universally applicable. Herein, applicant has also sought means to 
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1 automate the brightening process and thereby relieve the user, typically a novice, of 

2 the need to assess an ideal brightening function. 

3 An automatic strength selection process of the present invention can 

4 be applied to almost all photographs with little harm. The process is based on the 

5 assumption that there is an average strength value applicable to most photographs 

6 which is pleasing to the eye. Empirical study for this applicant has determined that 

7 there can be a target average strength which is suitable to most photographs. 

8 While this is based substantially on aesthetics, a range of target average strength 

9 can be usefully between 50 and 65 % of the dynamic range. A different target may 

10 be chosen by others, but nevertheless, an infinite number of different images can be 

1 1 corrected to a target average strength once selected. 

12 In correcting an image to a target average strength, applicant has 

13 avoided complex polynomial analysis where one might specify a different 

14 polynomial function for every image of an infinite possible number of images. 

15 Instead, while acknowledging that a preferred function for correcting a range of 

16 strengths present in an image can be a non-linear function, images are corrected to 

17 a more true-to-life color through simple interpolation of the results between a given 

18 function and a linear identity function (both of which are shown in Fig. 8) to 

19 determine what scaling factors will produce an image at the target strength value. 

20 Herein for simplicity of language, interpolation is intended to include extrapolation, 

21 both of which are merely estimates of a value from a function where other values of 

22 that function are known. 
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1 In an illustrative embodiment, an improved image enhancement 

2 process adapts the non-linear and continuous virtual iris curve to automatically 

3 obtain scaling factors for correcting the color if a digital image. Such a curve is 

4 shown in Fig. 8 as introduced in co-pending application 09/548,755. Other 

5 specialized curves illustrated in Figs. 4 and 6. The form and selection of the 

6 constants for the function are pre-determined, which does not adapt to the infinite 

7 variety of images. Applicant has determined a method for automatically adapting 

8 the function on an image by image basis. 

9 With reference to Fig. 19, herein, a 100% correction refers to direct 

10 application of a form of the virtual true color light amplification function to correct an 

1 1 image as shown as function C in Fig. 19 (also as Fig. 8). As stated, while the image 

12 is brightened and the color may remain true, this 100% correction may not provide 

13 an optimal aesthetic effect. Instead, an optimal or target correction has been found 

14 empirically to lie between the 100% correction and no correction at all, represented 

15 by function T. An identify function is a 0% correction, where no brightening is 

16 performed at all, illustrated as the diagonal linear line in Fig. 8. 

17 With reference also to Fig. 20, simply, for establishing the form of 

18 function T, a second original histogram Ho of the strengths of the dot maximum for 

19 each color dot in the original given image is obtained and an original average 

20 strength STRqrig is established. A third correction histogram He is obtained after a 

21 100% color amplification (function C) is applied to the original histogram and to 

22 establish a corrected average strength STRcorrI this correction being made 
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1 according to the invention so as ensure maintenance of the colors of the original 

2 given image and within the dynamic range. The corrected average strength 

3 STRcoRR is compared to a target average strength STRtar and scaling factors of 

4 the color correction are adjusted accordingly and while not a required step, a fourth 

5 histogram Ht can be established. 

6 Mathematical operations are preferably performed at the histogram 

7 level representing a set of values of individual primary colors R,G and B. Using a 

8 histogram, one scaling factor can determined which corresponds to one dot 

9 maximum which potentially represents a very large number of like color dots, all of 

10 which have the same dot maximum. This enables an image having millions of color 

1 1 dots to have a corresponding, manageable and computationally efficient 256 scaling 

1 2 factors representing the whole image. 

13 The original image may be as received or as previously adjusted using 

14 the faded image correction having been shifted by the offset x. 

15 In this embodiment, the first histogram Hs of a supplied image (Fig. 

16 18a) is shifted by offset x. A second and original histogram Ho (of dot maximums 

17 Fig. 18b) of this new original image can be determined as the beginning point for 

18 brightening the image. This new offset image forms the original image used for 

1 9 automatic strength selection as follows. 

20 With reference to Figs. 19 and 20, an average strength of the offset 

21 original image, STRqrig, can be calculated from the histogram Ho (Fig. 18b and Fig. 
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1 20). An Identity function applied to this histogram Ho would result in 0% correction 

2 and all the same strength values as in this original image. 

3 The virtual iris enhancement function C is then applied at the 100% 

4 correction for applying corresponding scaling factors selected from function C for 

5 each dot maximum of the histogram Ho, the scaling factor being applied to each of 

6 the R,G and B of each color dot so as to maintain the true color of the color dot from 

7 the original image. 

8 The third corrected histogram He can be created of the corrected 

9 image. Clearly, the fonnation of this and other histograms is purely a mathematical 

10 concept and no representation like Figs. 18a, 18b and 20 need be produced. A 

1 1 corrected average-strength, STRcorr. is determined from the values representing 

12 the corrected histogram He. 

13 Where the target average strength, STRtar, has been determined 

14 empirically, for example such as at 158 of a dynamic range of 255 (58%) we can 

15 determine whether the 100% correction was too aggressive and if a fractional 

16 correction should be applied. A fractional correction %CORR or interpolation 

17 constant can be determined through linear interpolation where, 

1 8 %CORR = (STRtar - STRoRir,) * 1 00%. 

19 (STRcoRR - STRorig) 

20 While the interpolation constant can be applied as directly calculated 

21 from the interpolation, the variability of images can result in unpredictable results. 

22 Accordingly, con-ection to the image can be constrained so that images are not 

51 



1 corrected more than 80% as a safeguard to the automatic process. Accordingly, for 

2 this embodiment and the function C set forth in Figs. 8 and 19, it is assumed that 

3 confection about 100% is too aggressive and should be bounded by about an 80% 

4 interpolation tempered with a lesser and fractional correction. In mathematical 

5 temis, the constraints are: 



6 %CORR = 100% correction - too aggressive 

7 thus 100% %CORR > 80% = 80% 

8 0% < %CORR 80% = %CORR 

9 %CORR 0% = 0% 

10 The virtual iris curve or function C is non-linear but the interpolated 



1 1 displacement from 100% of the virtual iris curve to the interpolated or target function 

12 T is a linear interpolation. It is in this sense that the linear interpolation is performed 

13 on a non-linear functions. 

14 

15 Example 

16 A brightly distorted and supplied image can be represented in a 

17 histogram Hs as in Fig. 18a. To determine the offset x, we take the value at 0.1% 

18 on the histogram, here x= 104 for a dynamic range of 0-255. The triplet (104, 104, 

19 104) is subtracted from each R, G, and B dot in the image and the second 

20 histogram Ho is created representing an original image, as in Fig. 18b and 

21 reproduced on Fig. 20. 
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1 Now the original image is briglitened without distortion of the colors. 

2 The mean, or average strength, of the original image, STRqrig. is found to be 95, 

3 being a fractional 0.37 or 37% of the dynamic range 

4 With reference to Fig. 19. the virtual Iris curve is applied to the original 



5 image. Assume there were 6400 color dots in the image having dot maximum 

6 strength values of strength 61 . Application of the identity function (0% connection) 

7 would maintain all 6400 values as a strength value of 61 , that is the image would be 

8 unchanged. When the virtual iris non-linear curve is applied to the image and a 

9 third correction histogram He is created, the 6400 values at an original strength 61 

10 are scaled to the value corresponding on the full or 100% virtual iris curve, say a n 

1 1 amplified or brightened strength value of 79. Only one scaling factor 0.79 / 0.61 = 

12 1 .295 need be calculated for each dot maximum which is applicable equally to each 

13 of R,G. and 6 in each of the 6400 dots in the image which all have the same 

14 strength. Similar adjustments are made to all strength levels within the dynamic 

15 range, scaling factors from the function C being applied to each dot maximum 

16 represented in the histogram Ho. 



17 From the correction histogram He of corrected strength values, a new 

18 mean, or corrected average-strength, STRcorr, is determined to be 170, being a 

19 fractional 0.67, or 67% of the dynamic range. We now know: 

20 STRoRiG = 95 (37%), STRcorr = 170 (67%) and STRtar = 158 (58%) 

21 To determine the automatic correction factor, %CORR, for the image 

22 we can apply the %CORR formula as follows: 
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1 % CORR = {(STRtar - STRorig) / (STRcorr - STRoRiG)riOO% 

2 %CORR = ((158 - 95) / (170 - 95)^100% 

3 %CORR = (63 / 75ri00 = 84% 

4 The interpolated constant %CORR represents the shift in the scaling 



5 factor between the identify function Z and function C. Accordingly, for the target 

6 correction, for all 6400 dots at strength 0.61 the appropriate scaling factor is no 

7 longer 1 .295 as determined at the 100% correction but is now an interpolated target 

8 scaling factor TSF which is established for each original dot maximum and is 

9 determined as follows: 



10 TSF = (%CORR*f D0tMaXrnpRPnTPn-D0tMaXnp»n.MA. H D0tMaXnp.n.MA. ) 

11 (100*DotMaxORIGINAL) 

12 where: DotMaxcoRRECTED represents the corrected dot maximum, and 

1 3 DotMaxoRiGiNAL represents the original dot maximum. 

14 Accordingly the target scaling factor TSF for dot maximums at 0.61 of 



1 5 the dynamic range is now ( 0.84 * (0.79-0.61 ) + 0.61 ) / 0.61 = 1 .248. Similar target 

16 scaling factors are determined for each dot maximum strength in the original image, 

17 adjusted to 84% of the 100% correction curve scaling factors. Accordingly, scaling 

18 factors at 84% of the Virtual Iris curve are applied to the original image, represented 

19 by function T, automatically creating an adjusted image having eye-pleasing color 

20 that is not distorted. 

21 This example image turns out be to already be a fairly bright image 

22 and one can further apply an arbitrary constraint to avoid overbrightening - judged 

23 solely on an esthetics basis, and thus where %CORR > 80%. then %CORR = 80% 
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1 This image corrected at 80% will result in an average strength in this 

2 case of 155. The scaling factors is therefore further suppressed slightly, the 

3 resulting scaling factor for a dot maximum at 61 being ( 0.80 * (0.79-0.61 ) + 0.61 ) / 

4 0.61 = 1.236. 

5 

6 Virtual Iris Functions Figs. 8.19 

7 The form of the function is specified empirically. The preferred virtual 

8 iris curve or function C of Fig. 8 and 19 is created by applying two separate non- 

9 linear functions to the no-correction line, shown here as an exponential and a 

10 logarithmic curve. Similarly, other functions include squared and square root 

11 functions. Other functions include suitably bounded polynomials and stepwise 

12 linear functions.. 

13 A pleasing connection is possible using the function or functions as set 

14 forth in Fig. 19 wherein a first exponential function CI ends where the second 

15 logarithmic function Cii begins. This is a predetermined anchor strength value xo 

16 intersecting the identify function or no-correction line Z and which is intermediate 

1 7 the minimum and the maximum of the dynamic range. The intersection xo has been 

18 determined empirically and for a dynamic range of 256 with the minimum at 0, Xo is 

1 9 set approximately as follows: 

20 Xo = 0 for a high quality image 

21 Xo = 2 for a clean image 

22 Xo = 25 for a noisy image 
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1 The first function CI is applied to the darkest area of the image 

2 between DRmin as the minimum of the dynamic range and the anchor strength value 

3 Xo, and which minimized an effect on the darkest area. The curve is exponential 

4 and has the form of f(x) = a + 10'"' where, DRmin x < xo, and f(DRmin ) = DRmm; 

5 f(xo)=xo. Typically DRmin is zero and the limits are 0 x < xo and accordingly f(0)=0. 

6 The second function Cii is applied to the brighter areas of the Image 

7 between the anchor strength value Xo and DRmax as the maximum of the dynamic 

8 range and has the effect of Imitating the iris so that the output image happens to 

9 more closely resemble one's visual memory of the experience. This curve is 

10 logarithmic and has the form of f(x) = a +logio(bx) where xO x < DRmax so that 

1 1 f(xo)=xo and f(DRmax) = DRmax- Typically a normalized DRmax is unity and the limits 

12 be(X)me xo x < 1 and accordingly f(1 )=1 . 

13 This invention has been described in detail with reference to particular 

14 embodiments thereof, but it will be understood that various other modifications can 

15 be applied within the spirit and scope of the invention. For example, different target 

16 average strengths may be preferred by others, different true color correction 

17 functions C can be applied as described herein, being linear, non-linear of 

18 combinations thereof. Further, many methodologies for computation analysis can 

19 be implemented to maximize efficiencies. Herein, histograms are applied to 

20 minimize the number of intensive calculations which is particularly important as 

21 dynamic range increases from 256, to 1024 and upwards, however, other less 
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1 efficient (color dot-by-dot) or other more efficient tecliniques can be applied and still 

2 achieve adjustment of an image without departing from the scope of the invention. 
3 



